Processes for monitoring software behavior have many applications in both offensive and defensive cyberwarfare and also in software testing/debugging. Whenever intelligence about an application needs to be gathered, the behavior of the application is monitored and conclusions drawn by analyzing the results.
When the source code of the application is available, one can monitor the behavior of the application on an instruction-by-instruction basis. Such monitoring allows the behavior to be better understood for the purposes of reengineering or debugging. Access to the source code allows exposure of all relevant information about the software's behavior, both external behavior and internal behavior. This includes calls to external components and the parameters of these calls, calls to internal components and the parameters of these calls, internally stored data, and control structures.
However, when the source code is not available, one must work with only the compiled binary of the application (for example, the executable, library or component). This means that only the machine code is available for analysis. As such it is much more difficult to get the detailed information that source debuggers can get.
Conventional systems include technology to intercept external behavior such as calls to the operating system or third party components. This technique is called system call interception. Conventional techniques also include the use of disassemblers to extract certain bits of internal information, but disassembly is a painstaking process fraught with trial and error. In general, information inside the binary executes without scrutiny and with no opportunity to intervene to change behavior in any predictable fashion. Internal functions, stored data, and individual control statements can be unreachable externally.
System call interception has many uses including proactive antivirus tools and testing/debugging tools. However, there are many behaviors that escape such external scrutiny. For example, instructions that call exception handlers or interrupts execute without making system calls and can bypass monitoring or protective software.
The only currently-known solution to intercept such dispatch mechanisms is to insert code into the operating system kernel that detects these dispatches and notifies the monitoring application. There is a disadvantage to this solution in that because the actual operating system kernel is modified, every single application is intercepted instead of just the application being monitored, and such “kernel mode” solutions tend to destabilize the operating system.
Therefore, there is a need for an improved software monitoring solution.